Fix silently incorrect scores in parea/evals (two inverted metrics, eval() on model output) - #1133

Description

Five eval functions in parea/evals/ return wrong scores without raising. Two of them are inverted — they report a perfect result as a failure and a total failure as perfect. Because these run inside _make_evaluations, which swallows exceptions and logs whatever number comes back, nothing surfaces the problem: the experiment completes and the dashboard shows a plausible score that is simply wrong.

Each fix is a separate commit, and each is covered by tests that fail against main (22 of the 39 new tests do).

1. answer_context_faithfulness_statement_level — score is inverted 🔴

answer_context_faithfulness_statement_level.py#L99

yes_count = sum(0 if "yes" in answer else 1 for answer in verdicts.strip().split(".") if answer != "")

The variable is named yes_count, but the ternary yields 0 when the verdict is "yes" and 1 otherwise, so it counts unsupported statements.

Driving the eval with a scripted grader response:

An answer entirely grounded in the retrieved context is scored as a complete hallucination.

The fallback on line 102, taken whenever the grader omits the Final verdict for each statement in order: summary, is wrong in two further ways:

return max(0, output.count("verdict: no")) / len(statements_formatted)

output is the answer being graded, not the grader's response, so the count is essentially always 0 — and it counts rejections as if they were supported statements. A fully supported answer whose grader reply lacked the summary line scored 0.0.

Fix: count "yes" verdicts, read the fallback from the grader's response, and clamp to 1.0 (a grader emitting more verdicts than there were statements could otherwise exceed 1).

2. context_ranking_listwise — NDCG is inverted 🔴

context_ranking_listwise.py#L107

return ndcg(reranked_indices, list(range(len(contexts))))

ndcg(y_true, ranking) expects y_true to be relevance grades, but reranked_indices is a permutation of context indices. The IDCG denominator is dcg(y_true, argsort(y_true)[::-1]), which is maximised when that index list is in descending order — i.e. when the reranker completely reverses the retrieved order.

All 24 permutations of 4 contexts, against sklearn.metrics.ndcg_score:

Pearson correlation between the reported score and the correct NDCG: −0.75. A perfect retriever scores 0.55; a retriever that returns its results in exactly the wrong order scores 1.0.

Fix: convert reranked_indices into relevance grades first — the context the reranker placed first gets the highest grade — then measure how well the retrieved order 0..n-1 agrees with them. Perfect retrieval now scores 1.0, and the regression test asserts that over every permutation of four contexts the score is uniquely maximised by the identity ranking and minimised by the reversed one.

Three further bugs on the same path:

• listwise_reranking cannot parse its own prompt's reply format. The prompt labels passages Passage1..PassageN, but parsing is [int(num) for num in s.split(",") if num.isdigit()]. A reply naming the passages (Passage3, Passage1, ...) parses to [], and contexts[offset:offset + window_size] = [] then silently deletes that window of contexts. A reply of bare numbers is 1-based but is used as 0-based indices, producing an off-by-one ranking and an IndexError at the window boundary. Now every integer in the reply is extracted, shifted to 0-based, bounds-checked and de-duplicated, with dropped passages appended — the result is always a permutation.
• n_contexts_to_rank=1 hangs forever. The factory validates n_contexts_to_rank >= 1, but window_step = n_contexts_to_rank // 2 is then 0, so offset -= window_step never terminates. Confirmed with a 10s alarm: no progress, and no API call is made inside the loop, so it spins on CPU indefinitely. Now max(1, n_contexts_to_rank // 2).
• dcg() overflows int64. gains = 2**rel - 1 on an integer array wraps for any relevance grade above 62 — dcg([70], [0]) returns -1.0, a negative gain. This is reachable now that grades scale with the number of retrieved contexts (a top-70 retrieval is ordinary). ndcg() also returned NaN rather than 0.0 when nothing was relevant.

3. percent_target_supported_by_context — eval() on model output 🔐

percent_target_supported_by_context.py#L84

match = re.search(pattern, classification.replace("\n", ""))
if match:
    response = eval(classification)

The grader's reply is passed to eval(), so whatever the model emits is executed as Python. A reply of

[{"Attributed": "Yes"}] and __import__("pathlib").Path("PWNED.txt").write_text("arbitrary code ran")

passes the regex and writes the file before any score is computed. Reachable by prompt injection through retrieved documents, which are attacker-controlled in most RAG deployments.

The regex is not a guard at all: it is applied to classification.replace("\n", "") while eval() receives the unmodified string, so the two never see the same text. That mismatch also breaks the ordinary case — a grader that pretty-prints its JSON or wraps it in a ```json fence raises SyntaxError straight out of eval().

Fix: parse the matched span with json.loads (via the existing safe_json_loads) and re.DOTALL so multi-line replies match. A missing Attributed key no longer raises AttributeError on None.

Unparseable replies now return None instead of 0.0. 0.0 was indistinguishable from "the context supports none of the target" and was logged as a real score; None skips the eval, which the return type already allowed. Happy to revert that part if you'd rather keep 0.0.

4. context_ranking_pointwise — one bad response makes the score NaN

context_ranking_pointwise.py#L80

response = [int("yes" in resp.get("verdict", " ").lower()) if resp.get("verdict") else np.nan for resp in response]

A verification that is not valid JSON, or that omits verdict, becomes np.nan, which then poisons both the numerator and the denominator:

all well-formed              -> 0.99999999995
one non-JSON reply           -> nan
one reply missing 'verdict'  -> nan

The NaN is logged as the score and silently destroys any mean computed over the experiment.

Fix: treat an unusable verification as "not relevant", matching RAGAS. numpy is no longer needed, so the import guard and its Raises: docstring entry are dropped.

5. balanced_acc — fractional scores truncated to zero

balanced_acc.py#L14

correct[log.target] += int(eval_result.score)

int() truncates toward zero, so a class whose scores were all 0.9 is reported as having 0.0 recall. Only an exact 1.0 counts. Now thresholded at >= 0.5.

This is the one judgement call in the PR — if you'd prefer balanced_acc to reject non-binary scores outright, say so and I'll change it.

Reproducing

Every case above is covered by tests/test_evals.py, which drives the eval functions with scripted grader responses via monkeypatch — no API calls, no keys, runs in ~0.3s.

poetry run pytest tests/test_evals.py -q     # 39 passed
git stash && poetry run pytest tests/test_evals.py -q  # 22 failed against main

(The n_contexts_to_rank=1 test hangs on main rather than failing, so deselect it when checking against the old code.)

The listwise ranking tests are written as properties rather than fixed numbers — the identity permutation must be the unique maximum and the reversed one the unique minimum — so they catch an inversion regardless of the exact gain formula.

Related Issue

Type of Change

• 📚 Examples / docs / tutorials / dependencies update
• 🔧 Bug fix (non-breaking change which fixes an issue)
• 🥂 Improvement (non-breaking change which improves an existing feature)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 🔐 Security fix
• 🆙 Version bump

Marked breaking because scores change — that is the point of the PR. Anything scored with the two inverted metrics needs re-running; historical values for them are not comparable to new ones.

Checklist

• I've read the CODE_OF_CONDUCT.md document.
• I've read the CONTRIBUTING.md guide.
• I've updated the code style using make codestyle.
• I've written tests for all new methods and classes that I created.
• I've written the docstring in Google format for all the methods and classes that I used.